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BACKGROUND 



10 Technical Field: 



The invention is related to incremental motion estimation techniques, and 
more particularly to an incremental motion estimation system and process for 
J estimating the camera pose parameters associated with each image of a long 

fi, 15 image sequence using a local bundle adjustment approach. 

ifl Background Art: 

O Estimating motion and structure of an object from long image sequences 

rv 2 0 depicting the object has been a topic of interest in the computer vision and the 
2 computer graphics fields for some time. For example, determining the motion 

^ and structure of objects depicted in a video of a scene containing the objects is 

of particular interest. The estimates are used for a variety of applications. For 
example, estimates of the structure of an object depicted in the consecutive 
25 images can be used to generating a 3D model of an object. Estimating the 
motion of an object in an image sequence is useful in background/foreground 
segmentation and video compression, as well as many other applications. A key 
part of estimating motion and structure from a series of images involves 
ascertaining the camera pose parameters associated with each image in the 
30 sequence. The optimal way to recover the camera pose parameters from long 
image sequences is through the use of a global bundle adjustment process. 
Essentially, global bundle adjustment techniques attempt to simultaneously 
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adjust the camera pose parameters associated with all the images such that the 
predicted 3D location of a point depicted in multiple images coincides. This 
typically involves the minimization of re-projection errors. However, bundle 
adjustment does not give a direct solution, rather it is a refining process and 
5 requires a good starting point. This starting point is often obtained using 
conventional incremental approaches. In general, incremental approaches 
attempt to estimate the camera pose parameters of an image in a sequence 
using the parameters computed for preceding images. 

10 A good incremental motion estimation process is also very important for 

applications other than just supplying a good starting point for the global bundle 
adjustment. For example, in many time-critical applications, such as visual 
navigation, there simply is not enough time to compute camera pose parameters 
using global bundle adjustment techniques, which are relatively slow processes. 

15 In such cases, the estimation of the camera pose parameters can be achieved 
using faster incremental approaches, albeit with potentially less accuracy. 
Additionally, when an image sequence is dominated by short feature tracks (i.e., 
overlap between successive images is small), the global optimization techniques 
degenerate into several weakly correlated local processes. In such cases, the 

20 aforementioned incremental methods will produce similar results with less 
processing. Still further, in some computer graphics applications, local 
consistency is more important than global consistency. For example, due to 
errors in calibration and feature detection, a global 3D model may not be good 
enough to render photorealistic images. Approaches such as "view-dependent 

2 5 geometry" which rely on a local 3D model may be preferable. Incremental 

methods can be employed as part of the process of generating these local 3D 
models. 

There are two main categories of incremental techniques. The first is 
30 based on Kalman filtering. Because of the nonlinearity between motion-structure 
and image features, an extended Kalman filter is typically used. The final result 
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then depends on the order in which the image features are supplied, and the 
error variance of the estimated motion and structure is usually larger than the 
bundle adjustment. 

5 The second category is referred to as subsequence concatenation. The 

present system and process falls into this category. One example of a 
conventional subsequence concatenation approach is the "threading" operation 
proposed by Avidan and Shashua in reference [1]. This operation connects two 
consecutive fundamental matrices using the tri-focal tensor. The threading 

10 operation is applied to a sliding window of triplets of images, and the camera 
matrix of the third view is computed from at least 6 point matches across the 
three views and the fundamental matrix between the first two views. Because of 
use of algebraic distances, the estimated motion is not statistically optimal. 
Fitzgibbon and Zisserman [2] also proposed to use sub-sequences of triplets of 

15 images. The difference is that bundle adjustment is conducted for each triplet to 
estimate the trifocal tensor and successive triplets are stitched together into a 
whole sequence. A final bundle adjustment can be conducted to improve the 
result if necessary. Two successive triplets can share zero, one or two images, 
and the stitching quality depends on the number of common point matches 

20 across six, five or four images, respectively. The number of common point 
matches over a sub-sequence decreases as the length of the sub-sequence 
increases. This means that the stitching quality is lower when the number of 
overlapping images is smaller. Furthermore, with two overlapping images, there 
will be two inconsistent camera motion estimates between the two images, and it 

25 is necessary to employ an additional nonlinear minimization procedure to 

maximize the camera consistency. It is also noted that both of the subsequence 
concatenation processes described above rely on point matches across three or 
more views. Point matches between two views, although more common, are 
ignored. 
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It is noted that in this background section and in the remainder of the 
specification, the description refers to various individual publications identified by 
a numeric designator contained within a pair of brackets. For example, such a 
reference may be identified by reciting, "reference [1]" or simply "[1]". A listing 
5 of the publications corresponding to each designator can be found at the end of 
the Detailed Description section. 

SUMMARY 

10 The present invention is directed toward an incremental motion estimation 

system and process that uses a sliding window of triplets of images, but unlike 
other subsequence concatenation processes, also takes into account those 
points that only match across two views. Furthermore, the motion is locally 
estimated in a statistically optimal way. To this end, a three-view bundle 

15 adjustment process is adapted to the incremental estimation problem. This new 
formulation is referred to as local bundle adjustment The problem is formulated 
as a series of local bundle adjustments in such a way that the estimated camera 
motions in the whole sequence are consistent with each other. Because the 
present technique makes full use of local image information, it is more accurate 

20 than previous incremental techniques. In addition, the present technique is very 
close to, and considerably faster than, global bundle adjustment. For example, 
in an experiment with 61 images, a reduction in processing time of almost 700 
times was observed while providing nearly the same accuracy. 

25 The present system and process begins by first extracting points of 

interest from each image in a given image sequence. This essentially involves 
determining the image coordinates of points associated with one or more objects 
of interest that are depicted in the image sequence. Any appropriate 
conventional process can be employed to determine the object point 

30 coordinates, such a corner detector. Once the points of interest are identified, 
point matches between successive images are established. Any appropriate 
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conventional technique can also be employed for this purpose. For example, 
sequential matchers have proven to be successful. In one embodiment of the 
present invention, two sets of point matches are gleaned from the point matching 
process. The first set identifies points that match in location in three successive 
5 images, while the second set identifies points that match in location between two 
successive images. Next, the camera coordinate system associated with the 
first image in the sequence is chosen to coincide with the world coordinate 
system. The motion (i.e., camera pose parameters) associated with the second 
image in the sequence is then computed using any appropriate conventional 
10 two-view structure-from-motion technique. This establishes the camera pose 
parameters (i.e., rotation and translation) of the first and second image with 
respect to the world coordinate system. 

A primary goal of the present system and process is to estimate the 
15 unknown camera pose parameters associated with the third image of a triplet of 
images in the given sequence using the parameters computed for the preceding 
two images. Thus in the case of the first three images, the camera pose 
parameters associated with the third image are estimated using the parameters 
previously computed for the first two images. This process is then repeated for 
20 each subsequent image in the sequence using the previously computed camera 
pose parameters from the preceding two images in the sequence. In this way, 
the camera pose parameters are estimated for each successive image in the 
sequence of images. The task of estimating the camera pose parameters 
associated with the third image in a triplet of images is accomplished using a 
25 process that will be described next. 

The aforementioned camera pose parameter estimation process 
according to one embodiment of the present invention involves minimizing the 
sum of the squared errors between the image coordinates of the identified match 
30 points and predicted coordinates for those points considering both the triple 
match points in a triplet of consecutive images and the dual match points 
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between the latter two images of the triplet. In one embodiment, the predicted 
image coordinates of a match point are computed as a function of the 3D world 
coordinate associated with the point under consideration and the camera pose 
parameters of the image depicting the point. More particularly, the camera pose 
parameters associated with the third image in a triplet of consecutive images of 
the given sequence, as well as the 3D world coordinates associated with each 
triple and dual match point in the triplet, which will result in a minimum value for 
the sum of the squared differences between the image coordinates of each 
identified triple match point and its corresponding predicted image coordinates in 
each of the three images in the triplet, added to the sum of the squared 
differences between the image coordinates of each identified dual match point 
and its corresponding predicted image coordinates in each of the latter two 
images in the triplet, are computed using the previously determined camera pose 
parameters associated with the first two images in the triplet. 

The use of the dual point matches in the foregoing technique provides a 
more accurate estimate of the camera pose parameters associated with each 
image in the image sequence than is possible using conventional incremental 
methods. In addition, the processing time associated with estimating the camera 
pose parameters of the images in the sequence is orders of magnitude faster 
than traditional global bundle adjustment techniques, while being nearly as 
accurate. However, it is possible to speed up the process even further. This is 
essentially accomplished by eliminating the need to directly estimate the 3D 
coordinates of the match points, and reducing the problem to one of minimizing 
just the camera pose parameters. Specifically, instead of estimating the 3D 
points, their projections into the images are estimated. To this end, the 
foregoing minimization problem is modified to characterize the predicted match 
point locations in terms of each other and functions that described the 
relationship between them. This is accomplished by recognizing each pair of 
match points in consecutive images must satisfy the epipolar constraint such that 
their locations are related by the fundamental matrix associated with the images 



in question. Additionally, in regard to triple match points, it is further recognized 
that given the image coordinates of the points in the first two images (which will 
have been computed previously), the location of the corresponding point in the 
third image can be defined in terms of the camera projection matrices (which 
5 contain the camera pose parameters) associated with the triplet of images. 
Applying these two constraints to the minimization technique described above 
allows the camera pose parameters associated with the third image in a triplet of 
consecutive images to be derived directly without having to estimate the 3D 
locations associated with the match points. This reduces the processing time 
10 significantly. 



DESCRIPTION OF THE DRAWINGS 



15 The specific features, aspects, and advantages of the present invention 

will become better understood with regard to the following description, appended 
claims, and accompanying drawings where: 

FIG. 1 is a diagram depicting a general purpose computing device 
20 constituting an exemplary system for implementing the present invention. 

FIGS. 2A and 2B are a flow chart diagramming an overall incremental 
motion estimation process for estimating the camera pose parameters 
associated with each image of a long image sequence. 

25 

FIG. 3A is a graph plotting errors in the translation magnitude against the 
image noise level for two embodiments of the process of Figs. 2A and 2B. 

FIG. 3B is a graph plotting errors in the rotation angle against the image 
30 noise level for two embodiments of the process of Figs. 2A and 2B. 
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FIG. 4 is a graph plotting the running time against the image noise level 
for two embodiments of the process of Figs. 2A and 2B. 

FIG. 5 is a ground truth comparison table comparing the mean rotation 
angle (a) between successive pairs of frames (in degrees), the sample 
deviation of the rotation angle (<tJ, the mean rotation axis (r ) in the order of x, 
y, and z, the mean translation vector (t), and the square roots of the diagonal 
elements of the covariance matrix of the translation (a ,) that results when 
various methods are used to estimate the camera pose parameters associated 
each image of a sequence of images. 

FIGS. 6A through 6D are images showing a visual comparison of the 
estimated camera motions estimated from the images of the STN31 sequence, 
where Method B was employed to estimate the camera motions depicted in Fig. 
6A, Method O was used to estimate the camera motions depicted in Fig. 6B, 
Method I was used to estimate the camera motions depicted in Fig. 6C and 
Method II was used to estimate the camera motions depicted in Fig. 6D. The 
small images with black backgrounds shown in each of the figures depict that 
portion where the start and end of the sequence meet, from a frontal direction. 
In addition, each plane in the figures represents a focal plane, of which the 
center is the optical center of the camera and the normal is the direction of the 
optical axis. 

FIGS. 7A and 7B are images showing a visual comparison of the camera 
motions estimated from the images of the STN61 sequence, where Fig. 7A 
shows a side and top view of the results obtained using a conventional global 
bundle adjustment technique, and Fig. 7B shows a side and top view of the 
results obtained using Method O. 

FIGS. 8A through 8C are images showing a visual comparison of the 
camera motions estimated from the images of the DYN40 sequence, where Fig. 
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8A depicts sample images from the input sequence, and Figs. 8B and 8C each 
show top and front views of the results obtained using global bundle adjustment 
and Method O, respectively. 

FIGS. 9A through 9C are images showing a visual comparison of the 
camera motions estimated from the images of the FRG21 sequence, where Fig. 
9A shows a top view of the results obtained using Method O, and Fig. 9C is a top 
view of the results obtained using global bundle adjustment. Fig 9B shows a 
sample image of the sequence. 

FIG. 10 is a table comparing the projection error and running time 
obtained using various methods (i.e., I, II, O and B) on four sample image 
sequences (STN31, STN61, DYN40, and FRG21). The projection errors are 
displayed in the top of each cell of the table. The error used here is the root 
mean square error in pixels. The running time (in seconds) of each algorithm is 
displayed under each corresponding projection error. The numbers below the 
sequence names are the estimated average rotation angle between two 
successive frames. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following description of the preferred embodiments of the present 
invention, reference is made to the accompanying drawings which form a part 
hereof, and in which is shown by way of illustration specific embodiments in 
which the invention may be practiced. It is understood that other embodiments 
may be utilized and structural changes may be made without departing from the 
scope of the present invention. 

Before providing a description of the preferred embodiments of the 
present invention, a brief, general description of a suitable computing 



environment in which the invention may be implemented will be described. 
Figure 1 illustrates an example of a suitable computing system environment 100. 
The computing system environment 100 is only one example of a suitable 
computing environment and is not intended to suggest any limitation as to the 
scope of use or functionality of the invention. Neither should the computing 
environment 100 be interpreted as having any dependency or requirement 
relating to any one or combination of components illustrated in the exemplary 
operating environment 100. 

The invention is operational with numerous other general purpose or 
special purpose computing system environments or configurations. Examples of 
well known computing systems, environments, and/or configurations that may be 
suitable for use with the invention include, but are not limited to, personal 
computers, server computers, hand-held or laptop devices, multiprocessor 
systems, microprocessor-based systems, set top boxes, programmable 
consumer electronics, network PCs, minicomputers, mainframe computers, 
distributed computing environments that include any of the above systems or 
devices, and the like. 

The invention may be described in the general context of computer- 
executable instructions, such as program modules, being executed by a 
computer. Generally, program modules include routines, programs, objects, 
components, data structures, etc. that perform particular tasks or implement 
particular abstract data types. The invention may also be practiced in distributed 
computing environments where tasks are performed by remote processing 
devices that are linked through a communications network. In a distributed 
computing environment, program modules may be located in both local and 
remote computer storage media including memory storage devices. 

With reference to Figure 1, an exemplary system for implementing the 
invention includes a general purpose computing device in the form of a computer 
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110. Components of computer 110 may include, but are not limited to, a 
processing unit 120, a system memory 130, and a system bus 121 that couples 
various system components including the system memory to the processing unit 
120. The system bus 121 may be any of several types of bus structures 
including a memory bus or memory controller, a peripheral bus, and a local bus 
using any of a variety of bus architectures. By way of example, and not 
limitation, such architectures include Industry Standard Architecture (ISA) bus, 
Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video 
Electronics Standards Association (VESA) local bus, and Peripheral Component 
Interconnect (PCI) bus also known as Mezzanine bus. 

Computer 1 10 typically includes a variety of computer readable media. 
Computer readable media can be any available media that can be accessed by 
computer 1 10 and includes both volatile and nonvolatile media, removable and 
non-removable media. By way of example, and not limitation, computer 
readable media may comprise computer storage media and communication 
media. Computer storage media includes both volatile and nonvolatile, 
removable and non-removable media implemented in any method or technology 
for storage of information such as computer readable instructions, data 
structures, program modules or other data. Computer storage media includes, 
but is not limited to, RAM, ROM, EEPROM, flash memory or other memory 
technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, 
magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic 
storage devices, or any other medium which can be used to store the desired 
information and which can be accessed by computer 110. Communication 
media typically embodies computer readable instructions, data structures, 
program modules or other data in a modulated data signal such as a carrier 
wave or other transport mechanism and includes any information delivery media. 
The term "modulated data signal" means a signal that has one or more of its 
characteristics set or changed in such a manner as to encode information in the 
signal. By way of example, and not limitation, communication media includes 
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wired media such as a wired network or direct-wired connection, and wireless 
media such as acoustic, RF, infrared and other wireless media. Combinations of 
the any of the above should also be included within the scope of computer 
readable media. 

The system memory 130 includes computer storage media in the form of 
volatile and/or nonvolatile memory such as read only memory (ROM) 131 and 
random access memory (RAM) 132. A basic input/output system 133 (BIOS), 
containing the basic routines that help to transfer information between elements 
within computer 1 10, such as during start-up, is typically stored in ROM 131 . 
RAM 132 typically contains data and/or program modules that are immediately 
accessible to and/or presently being operated on by processing unit 120. By way 
of example, and not limitation, Figure 1 illustrates operating system 134, 
application programs 135, other program modules 136, and program data 137. 

The computer 110 may also include other removable/non-removable, 
volatile/nonvolatile computer storage media. By way of example only, Figure 1 
illustrates a hard disk drive 141 that reads from or writes to non-removable, 
nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes 
to a removable, nonvolatile magnetic disk 152, and an optical disk drive 155 that 
reads from or writes to a removable, nonvolatile optical disk 156 such as a CD 
ROM or other optical media. Other removable/non-removable, 
volatile/nonvolatile computer storage media that can be used in the exemplary 
operating environment include, but are not limited to, magnetic tape cassettes, 
flash memory cards, digital versatile disks, digital video tape, solid state RAM, 
solid state ROM, and the like. The hard disk drive 141 is typically connected to 
the system bus 121 through an non-removable memory interface such as 
interface 140, and magnetic disk drive 151 and optical disk drive 155 are 
typically connected to the system bus 121 by a removable memory interface, 
such as interface 150. 
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The drives and their associated computer storage media discussed above 
and illustrated in Figure 1, provide storage of computer readable instructions, 
data structures, program modules and other data for the computer 110. In 
Figure 1, for example, hard disk drive 141 is illustrated as storing operating 
5 system 144, application programs 145, other program modules 146, and 

program data 147. Note that these components can either be the same as or 
different from operating system 134, application programs 135, other program 
modules 136, and program data 137. Operating system 144, application 
programs 145, other program modules 146, and program data 147 are given 

10 different numbers here to illustrate that, at a minimum, they are different copies. 
A user may enter commands and information into the computer 110 through 
input devices such as a keyboard 162 and pointing device 161 , commonly 
referred to as a mouse, trackball or touch pad. Other input devices (not shown) 
may include a microphone, joystick, game pad, satellite dish, scanner, or the like. 

15 These and other input devices are often connected to the processing unit 120 
through a user input interface 160 that is coupled to the system bus 121 , but 
may be connected by other interface and bus structures, such as a parallel port, 
game port or a universal serial bus (USB). A monitor 191 or other type of display 
device is also connected to the system bus 121 via an interface, such as a video 

20 interface 190. In addition to the monitor, computers may also include other 

peripheral output devices such as speakers 197 and printer 196, which may be 
connected through an output peripheral interface 195. Of particular significance 
to the present invention, a camera 163 (such as a digital/electronic still or video 
camera, or film/photographic scanner) capable of capturing a sequence of 

25 images 164 can also be included as an input device to the personal computer 
110. Further, while just one camera is depicted, multiple cameras could be 
included as input devices to the personal computer 110. The images 164 from 
the one or more cameras are input into the computer 110 via an appropriate 
camera interface 165. This interface 165 is connected to the system bus 121, 

30 thereby allowing the images to be routed to and stored in the RAM 132, or one of 
the other data storage devices associated with the computer 110. However, it is 
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noted that image data can be input into the computer 110 from any of the 
aforementioned computer-readable media as well, without requiring the use of 
the camera 163. 

The computer 110 may operate in a networked environment using logical 
connections to one or more remote computers, such as a remote computer 180. 
The remote computer 180 may be a personal computer, a server, a router, a 
network PC, a peer device or other common network node, and typically includes 
many or all of the elements described above relative to the computer 110, 
although only a memory storage device 181 has been illustrated in Figure 1. 
The logical connections depicted in Figure 1 include a local area network (LAN) 
171 and a wide area network (WAN) 173, but may also include other networks. 
Such networking environments are commonplace in offices, enterprise-wide 
computer networks, intranets and the Internet. 

When used in a LAN networking environment, the computer 110 is 
connected to the LAN 171 through a network interface or adapter 170. When 
used in a WAN networking environment, the computer 110 typically includes a 
modem 172 or other means for establishing communications over the WAN 173, 
such as the Internet. The modem 172, which may be internal or external, may 
be connected to the system bus 121 via the user input interface 160, or other 
appropriate mechanism. In a networked environment, program modules 
depicted relative to the computer 1 10, or portions thereof, may be stored in the 
remote memory storage device. By way of example, and not limitation, Figure 1 
illustrates remote application programs 185 as residing on memory device 181. 
It will be appreciated that the network connections shown are exemplary and 
other means of establishing a communications link between the computers may 
be used. 
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The exemplary operating environment having now been discussed, the 
remaining part of this description section will be devoted to a description of the 
program modules embodying the invention. 

1. Local Bundle Adjustment 

In this section, the notation that will be used to describe the present local 
bundle adjustment technique will be introduced, and the technique itself will be 
described. 

1.1. Notation 

An image point is denoted by p = [u, v] T , and a point in space is denoted 
by P = [X, V, Z] r . For homogeneous coordinates, x = [x r , 1] T is used for any 
vector x. Image I t is taken by a camera with unknown pose parameters M, which 
describes the orientation (rotation matrix R/) and position (translation vector t) of 
the camera with respect to the world coordinate system in which 3D points are 
described. The relationship between a 3D point P and its projection p, in image / 
is described by a 3 x 4 projection matrix P,, and is given by 

spt = P*P , (1) 

where s is a non-zero scale factor. In general, P, = A,[R, %], where the 3 x 3 
matrix A, contains the camera's internal parameters. If the camera is calibrated 
(i.e., A, is known), normalized image coordinates are employed and A, is set to 
the identity matrix. In the following discussion, it is sometimes more convenient 
to describe the non-linear projection (Eq. 1) by function <fr such that 

p,= 0(M„ P). (2) 
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1.2. Local Bundle Adjustment 

Referring to Figs. 2A and 2B, the local bundle adjustment technique 
according to the present invention will now be described. In process action 200, 
5 an image sequence {h | / = 0,...,N - 1} is input. Points of interest in the 

sequence are extracted from each image (process action 202). Any appropriate 
extraction procedure can be employed for this purpose. For example, a Harris' 
corner detector [3] was used in tested embodiments of the present system and 
process. Point matches between successive images are then established 

10 (process action 204). Any appropriate conventional matching technique can be 
used. For example, standard sequential matching techniques [4, 5, 6, 7] would 
be appropriate for use with the present invention. Once the point matches are 
known, they are separated into two categories (at least in one embodiment of the 
present invention), as indicated in process action 206. The first category 

15 contains point matches across all three views, and the second category contains 
point matches only between the last two images of the triplet under 
consideration. The first image 7 0 in the input sequence is then selected and its 
camera coordinate system is designated as the 3D "world" coordinate system 
(process action 208). Thus, R 0 = I and to = 0. Next, as indicated by process 

20 action 210, the second image (i.e., h ) in the input sequence is selected and a 
standard two-view structure-from-motion technique, such as one based on 
minimizing re-projection errors, is used to compute the motion Mi (i.e., camera 
pose parameters). The next previously unselected image (i.e., h (/> 2)) in the 
input sequence is then selected (process action 212). The motion M, associated 

25 with the last-selected image is then estimated (process action 214). This is 

accomplished using the triplet of images including the last-selected image and 
the two immediately preceding images (i.e., £_ 2 , //), and the point matches 
between only the last two images of the triplet as well as the point matches 
across all three images of the triplet, as will be described in detail shortly. It is 

30 next determined if the last-selected image is the last image in the input image 
sequence (process action 216). If it is the last image, the process ends. 
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However, if there are remaining previously unselected images in the sequence, 
Process actions 212 through 216 are repeated. 

The process by which M, , where /> 2, is estimated will now be described. 
To simplify the explanation, only / = 2 will be considered, however, the result 
extends naturally to / > 2. Consider the three views (4 h, h)- Two sets of point 
matches have been identified. The first contains point matches across all three 
views, and is denoted by O = {(p 0 , ; , Pi, 7 , P2,y ) I j = 1 , • • • , M}. The second 
contains point matches only between h and J 2 , and is denoted by O = {(qi,/c, <\2,k 
) | k = 1,..., N}. The camera matrices P 0 and Pi (or equivalently M 0 and Mi) are 
already known, and the problem is to determine the camera matrix P2 (or 
equivalently M 2 ). 

The objective is to solve for P2 orM 2 in an optimal way by minimizing 
some statistically and/or physically meaningful cost function. A reasonable 
assumption is that the image points are corrupted by independent and identically 
distributed Gaussian noise because the points are extracted independently from 
the images by the same algorithm. In that case, the maximum likelihood 
estimation is obtained by minimizing the sum of squared errors between the 
observed image points and the predicted feature points. More formally, the 
process for estimating M 2 according to the present invention is: 



where P, is the 3D point corresponding to triple point match (p 0 , y , pi,;, Pzy ) and 
Q k is the 3D point corresponding to pair point match (q 1>fc , q 2 ,k )■ It should be 
noted that this process advantageously includes point matches shared by two 




(3) 
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views as well as those across three views to achieve a more accurate estimate 
than possible with conventional techniques. In addition, this process guarantees 
that the estimated consecutive camera motions are consistent with a single 3D 
model defined in the camera coordinate system of the first image of the 
5 sequence. 

It is noted that in connection with estimating Mi (see process action 210), 
when the motion between k and h is small, a standard two-view structure-from- 
motion technique might result in an inaccurate estimate for the camera pose 
10 parameters. In such cases, the pose parameters Mi associated with the second 
image h can be more accurately estimated using a conventional global bundle 
adjustment procedure performed on the first three images of the sequence 4 h 
and 7 2) albeit at the cost of a longer processing time. 

15 It is further noted that in cases where one or more of the internal camera 

parameters, such as for example the focal length, are not known, the foregoing 
technique can be modified to determine the projection matrix that will provide the 
minimum difference between the observed match point locations and the 
predicted locations. The projection matrix includes both the internal and 

2 0 extrinsic parameters, whereas the camera pose parameters only include the 

rotation and translation coordinates associated with a camera (i.e., the extrinsic 
parameters). In this way the unknown internal parameters can be determined as 
well. Even when the internal parameters are known, the projection matrix can be 
determined in lieu of the camera pose parameters, if desired. However, the 

25 elimination of the internal parameters reduces the processing time somewhat. 

2. Reducing The Local Bundle Adjustment Processing Time 

The minimization process of Eq. 3 is characterized by a large dimensional 
30 space, with the number of dimensions being equal to 6 + 3(M + A/). While even 
with this large dimension space much less processing is required compared to 
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global bundle adjustment procedures, the process could still be made much 
quicker. One way to speed up the minimization process would be to employ the 
sparseness of the Jacobian and Hessian structure as described in reference [8]. 
However, a specialized procedure for speeding up the processing associated 
with the present local bundle adjustment technique has been developed and will 
be described next. 

The aforementioned procedure involves the use of a first order 
approximation to eliminate all the unknown structure parameters {P,} and {Q*}, 
thus reducing the local bundle adjustment to a minimization problem over only 
the camera pose parameters M 2 (a six-dimensional space if the camera's 
internal parameters are known). 

Because of independence between the 3D point structures, Eq. (3) is 
equivalent to 



The above two terms will be considered separately in the sections to follow. 

2.1. Two-view and Three-view Geometry 

Before describing the process for reducing the local bundle adjustment to 
a minimization problem over only the camera pose parameters M 2 , some new 
notation regarding two-view and three-view geometry must be introduced. 
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2.1 .1 . Epipolar constraint In order for a pair of points (p,- , p /+1 ) between I t and 
to be matched (or in other words to correspond to a single point in space), 

they must satisfy the epipolar constraint: 

where the fundamental matrix F i;in is given by: 

Fm+i = [P*+i c i] x P^-hiPi" « (6) 

10 

Here, c, is a null vector of P/, i.e., P/c/= 0. Therefore, c/ indicates the position of 
the optical center of image P + is the pseudo-inverse of matrix 
P: P + = P r (PP r )" 1 . [x] x denotes the 3 x 3 anti-symmetric matrix defined by vector 
x such that x x y = [x] x y for any 3D vector y. When the cameras 1 internal 
15 parameters are known, it is possible to work with normalized image coordinates, 
and the fundamental matrix becomes the essential matrix. Thus, F^i = [tu+i] x 
R^h-i, where (R/, /+ i, t /fI+ i) is the relative motion between h and I M . 

2.1.2. Image point transfer across three views . Given the camera projection 
20 matrices P, (/ = 0, 1 , 2) and two points p 0 and p^ 1 in the first two images 

respectively, their corresponding point in the third image, P2, is then determined. 
This can be seen as follows. Denote their corresponding point in space by P. 

According to the camera projection model (Eq. 1), s 0 p 0 =P 0 Pand 5 l p 1 =P t P . 

Eliminating the scale factors yields: 

25 

[p G ]xPoP = 0 and |pi] x PiP = 0. 



It is easy to solve for P from these equations. Then, its projection in h is given 
by S2P2 = P 2 P ■ Combining the above two operations gives the image transfer 
function which we be denoted by y/, i.e., 



P2= y(Po, pi) 



(7) 



10 



2.1 .3. Additional notation . The following notation is also introduced to facilitate 
the explanation of the process for reducing the local bundle adjustment to a 
minimization problem over only the camera pose parameters M2. Specifically, it 
is noted that: 



P = 



and Z = 



1 0 
0 1 
0 0 



(8) 
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Thus, p = Zp, p = Z r p, ZZ r =diag (1,1, 0), and Z 7 Z = diag (1,1). 
2.2. Simplifying the Three-View Cost Function 



Consider 
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' i=0 



(9) 
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in Eq. (4). To simplify the notation, the subscript j is dropped. Instead of 
estimating the 3D point, its projections in images is estimated. This is equivalent 
to estimating p, (/ = 0, 1 , 2) by minimizing the following objective function: 
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J = llPo - Poll 2 + liPi - Pif + ||P2 - P2II 2 (10) 



subject to (i) p x F 0) ip 0 = 0 (11) 
and(ii) P2 = ^(pojPi) , (12) 



where the fundamental matrix F 0 ,i and the image transfer function y/ are 
described in Sec. 2.1 . By applying the Lagrange multiplier technique, the above 
5 constrained minimization problem can be converted into an unconstrained 
minimization problem with the new objective function given by: 

J f = J + Ai^ + A 2 T (13) 

10 where 

~t ~ 

^ = PiFo,iPo andT= ||p 2 - ^(po, Pi) IP- 
Define A pi (/= 0, 1,2) as 

15 Ap^pi-pi, (14) 

then 

Pi = Pi - Apt, or pj ^ Pi - Api. 
Referring to Eq. (8) for p. 
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If the second order term is neglected, the constraint of Eq. (1 1 ) becomes 
5 = 0, with 

F w pf F 0 ,iPo - Apf Fo.ipo - pf F 0 ,iApo 
= Pf F 0; iPo - Apf Z T F 0 ,iPo - pf F 0 ,iZApo . (15) 



25 Here, the equation p = Zp is used. The constraint of Eq. (12) becomes 
T = ||p 2 - Ap 2 -^(p 0 - A Po ,pi - A Pl )|| 2 (16) 
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By applying a Taylor expansion and keeping the first order terms: 



V>(po-Ap 0) pi-Api) «^(Po>Pi)-*oAp 0 -#iApi (17) 

where 

** = 5^ (po,Pi) /^Pi- 
Equation (16) then becomes T= 0, with 



T « ||p 2 - Ap 2 - (po,Pi) + *oAp 0 + #iApi|| 2 
& a T a + 2a T * 0 Ap 0 + 2a T *i Ap x - 2a T Ap 2 (18) 



where a = P2- ^(Po, Pi), and the second approximation in Eq. (18) is achieved 
by only keeping first order items. 

To minimize f in Eq. (13), let its first-order derivatives with respect to A p ; 
be zero, which yields 



= 2A P0 - XiZ 1 F^pi + 2A 2 *6 a 

8 7* 

= 2A Pl - A 1 Z T F 0| ip 0 + 2A 2 *f a 

^— = 2Ap 2 - 2A 2 a . 
aAp 2 



This gives 



Apo = ^AiZ^F^p! - A 2 *Ja (19) 

A Pl = ^AxZ^Fo.iPo - A 2 #fa (20) 
Ap 2 = A 2 a . (21) 



Substituting them into the linearized constraints of Eqs. (15) and (18) gives 



9i ~ Ai£ 2 /2 + X293 = 0 (22) 
94 + Ai#j/2 - X295 = 0 , (23) 



where 



91 


= pfFo.iPo 


(24) 


92 


= pjfFj^ZZ^FoaPo +pfF 0jl ZZ ;r Fj 1 p 1 


(25) 


93 


= a^Z^ciPo +pfF 0( iZ^a 


(26) 


94 


1 T 
= -a a 


(27) 


95 


= a r (I + # 0 «f + *i*f)a. 


(28) 



T 10 



Thus, the solution for X1 and X2 is 



Ai = g gWi+gltt (29) 

0205 - 9$ 

, 0204 + 9193 „„s 

A 2 = o— . (30) 

0205 - 93 



15 Substituting the obtained 7n, A-2, A po, A pi and A P2 back into Eq. (13) finally 
gives 



J' = | A f 32 - AiA 2 £3 + \\g 5 
_ 0105 + 0204 + 2919394 

9295 - gl 



(31) 
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This is the new cost functional for a point match across three views. It is a 
function of the motion parameters, but does not contain 3D structure parameters 
anymore. 

2.3. Simplifying the Two-View Cost Function 
Consider now 

2 

mmT\\qi, k -<p(niM\\ 2 (32) 

1=1 

in Eq. (4). To simplify the notation, the subscript k is dropped. Following the 
same idea as in the last subsection, this is equivalent to estimating q f and q 2 by 
minimizing the following objective function 

£=||qi-qill 2 + llq2-q 2 || 2 (33) 

subject to q^Fi^qx^O (34) 
where the fundamental matrix Fi j2 is defined as in section 2.1. 

Define Aq., -q 1 and Aq 2 =q 2 -q 2 . Linearize the constraint of Eq. 
(34) as in Eq. (15). Using the Lagrange multiplier technique, the above 
constrained minimization problem can be transformed into an unconstrained 
one: 

£' = Aqf Aqi + Aq^ Aqs + (35) 
A(qfF 1>2 qi - AqJZ^i^q! -qfFx^ZAq!) . 

Letting the first-order derivatives be zero gives: 




qfF?; a ZZ3'Fi |2 qi+S , Fi, 2 ZZ2 , Ff ia q 2 ' 



Substituting them back into Eq. (35), after some simple algebra, gives: 



This is the new cost functional for a point match only between two views. It is a 
function of the motion parameters, but does not contain any 3D structure 
parameters. 

2.4. Summary 

In summary, Eq. (4) becomes the following minimization problem: 



where f and L' are given by Eqs. (31) and (36). Notice that, using a first order 
approximation, the original large problem is transformed into a minimization over 
just the dimensional camera pose space. 

In one embodiment of the present invention, the above technique is 
implemented using the conventional Levenberg-Marquardt algorithm. The initial 
guess at the minimum camera pose is obtained as follows. Only point matches 
across three views are used, and they are reconstructed in 3D space using 
points between the first and second view because their camera poses are 



(q?Fi,2qi) 2 



(36) 



qTF^ZZ^q^+qlFx.sZZrF^ ' 




(37) 



known. Then, the camera pose for the third view is computed using 3D-2D 
correspondences. In both stages, a linear solution is started with, followed by a 
refining based on the distances between the observed image points and the 
predicted feature points. 

3.0 Experimental Results 

In this section, we provide experimental results with both synthetic (Sec. 
3.1) and real data (Sec. 3.2 through Sec. 3.4). We consider a number of 
methods: 

a) Method L: An implementation of the exact local bundle adjustment 
of Eq. (3) or more precisely Eq. (4), in which the independency of point 
structures (or equivalently the sparseness in the Jacobian and Hessian matrices) 
has been taken into account. 

b) Method 0: Our reduced local bundle adjustment of Eq. (37), 
where structure parameters have been eliminated. 

c) Method I: The algorithm used to initialize our local bundle 
adjustment, as described in Sec. 2.4. Only matches shared by three views are 
used. This is similar to the approach of reference [1], except the re-projection 
errors, instead of algebraic errors, are used. 

d) Method II: A two-view incremental algorithm. Motion is estimated 
only from point matches between the last two images, and the undetermined 
scale is computed using the common matches shared with those between the 
previous image pair. 

e) Method B: A global bundle adjustment technique which gives the 
statistically optimal solution. The independency of point structures has been 
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taken into account. The result obtained with Method 0 is used to initialize this 
method. 

The above algorithms have been implemented with VC++, and all 
5 experiments reported in this section were conducted on a PC with a Pentium® III 
850 MHz processor. 

The synthetic data was used to compare Method 0 and Method L. It was 
generated from real data composed of a 3-image subsequence (i.e., the first 3 
10 images of the STN31 sequence to be described shortly) in the following way. 
First, we reconstructed the 3D point for each track with the real feature points 
and camera motions. Second, original feature points in a track were replaced by 
the projected 2D points of the reconstructed 3D points. After this replacement, 
the camera motions and the 3D points in the real data became our ground truth. 

15 

Four real sequences named STN31, STN61, DYN40, and FRG21 were . 
used in the experiments. STN61 is a 61 -image sequence of a stone sculpture 
on a turntable, of which the rotation angle can be accurately controlled. STN31 
is a 31 -image sequence subsampled from the original STN61 sequence. Both 

20 STN31 and STN61 are closed-matches loop sequences, meaning that the first 
image and the last images coincide (i.e., their motion is zero). DYN40 is a 40- 
image sequence taken with a multi-camera rig. FRG21 is a 21 -frame sequence 
taken by a hand held camera targeting a toy frog. The camera intrinsic 
parameters for STN31, STN61 , and FRG21 were calibrated in advance, and 

25 those for DYN40 were self-calibrated and thus less accurate. While the 

algorithms tested in this section do not require any special motion sequences, 
we intentionally chose orbital or near orbital ones in order to make the 
experiment results easily interpretable. 

30 With the real data, we compared methods O, I, II, and B. Section 3.2 

gives detailed quantitative and visual comparisons of the four methods with help 
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of some of STN31's properties. Section 3.3 shows the visual comparisons of 
Method O and Method B with the other three sequences. Section 3.4 presents 
the quantitative comparisons of the four methods in terms of the projection errors 
and running time. 

5 

3.1. Synthetic Data 

We compared Method L and Method O with the synthetic data. There are 
about 200 points and the interframe rotation angle is about 12 degrees. The two 

10 methods are used to compute the third camera motion under different image 
noise levels. We used the relative rotation angle and the translation magnitude 
with respect to the ground truth as the error metrics of the estimated motion. At 
each image noise level, we run both methods 30 times, and the mean 
differences between the computed motion and the ground truth motion were 

15 recorded. Average running time was recorded in a similar way. Figures 3A and 
3B show the translation magnitude errors and the rotation errors, respectively, 
for both methods. We can see from the figure that the errors with Method O are 
only slightly higher than Method L. It is less than 0.2% larger even when noise 
with 1 pixel standard deviation was added. On the other hand, this accuracy has 

20 been achieved with 30 times less computational time as can be seen from Fig. 
4. This shows that the mathematical procedure described in Sec. 2 to eliminate 
all structure parameters is indeed of benefit. 

3.2. Ground Truths Comparisons 

25 

The STN31 sequence is used next. There are several things we know for 
sure, within the control accuracy offered for our turntable, about the STN31 
sequence. First, the camera is moving on a circle and the camera motion of the 
first image is overlapped with the last one. Second, the relative rotation angle 
30 and translation vector between two consecutive images are the same 

throughout the sequence. We also roughly know that relative rotation angle is 
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around 12.414 degrees, and the rotation axis is close to [0, 1, 0]. The results are 
listed in the ground truth comparison table shown in Fig. 5, where a is the mean 
rotation angle between successive pairs of frames (in degrees), crjs the sample 
deviation of a, r is the mean rotation axis (in the order of x, y, and z), t is the 
5 mean translation vector, and a t are the square roots of the diagonal elements of 
the covariance matrix of the translation. It can be seen from the table that the 
proposed method is superior to both Method I and Method II in terms of the 
accuracy of both rotation angle and translation vector, e.g., it has smaller a a and 
a f. On the other hand, the results of our method are in general very close to 
10 those given by the global bundle adjustment method. 

Figures 6A through 6D show the visual comparison of the estimated 
camera motions for the STN31 sequence, where Method B was employed to 
estimate the camera motions depicted in Fig. 6A, Method O was used to 

15 estimate the camera motions depicted in Fig. 6B, Method I was used to estimate 
the camera motions depicted in Fig. 6C and Method II was used to estimate the 
camera motions depicted in Fig. 6D. The small images with black backgrounds 
shown in each of the figures depict that portion where the start and end of the 
sequence meet, from a frontal direction. In addition, each plane in the figures 

2 0 represents a focal plane, of which the center is the optical center of the camera 
and the normal is the direction of the optical axis. The absolute size of the plane 
is not meaningful and has been properly adjusted for display purposes. 
According to the first ground truth mentioned earlier, the focal plane arrays 
should form a closed circle if the camera motions are computed correctly. This 

25 conforms quite well with the results of both our method and the bundle 

adjustment, with the exception that the overlapping between the first focal plane 
and the last one is a little bit overhead (see the small black images in the figure 
for details). This reveals that accumulation error is inevitable in both cases. 
Note that we did not use the knowledge that the last image is the same as the 
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first image. Obviously, the results with the other two methods are much less 
accurate. 

3.3. Visual Comparisons with Bundle Adjustment 

This subsection concentrates on the visual comparison of our method 
against the results of the global bundle adjustment procedure using the 
remaining three real image sequences. In brief, Figs. 7A and 7B, Figs. 8A 
through 8C, and Figs. 9A through 9C show the results of STN61, DYN40, and 
FRG21 , respectively. Specifically, Fig. 7A shows a side and top view of the 
results obtained using global bundle adjustment, and Fig. 7B is a side and top 
view of the results obtained using Method O, employing the STN61 sequence as 
an input. Fig. 8A depicts sample images from the DYN40 input sequence, and 
Figs. 8B and 8C each show top and front views of the results obtained using 
global bundle adjustment and Method O, respectively. Fig. 9A shows a top view 
of the results obtained using Method O, and Fig. 9C is a side and top view of the 
results obtained using global bundle adjustment, employing the FRG21 
sequence as an input. Fig 9B shows a sample image of the sequence. 
Together with Figs. 6A through 6D, the foregoing figures will be used as the 
complementary results for the quantitative comparisons in the next subsection. 

3.4. Comparisons on Projection Errors and Running Time 

We now compare the different methods in terms of the projection errors 
and running time. The projection errors were computed from a tracked feature 
table and the camera motions. The tracked feature table contains a list of 
feature tracks. Each track is a list of 2D point features that are shared by 
different views. The camera motions were computed with four different methods 
(i.e., I, II, O and B) as described above. The camera motions were used to 
reconstruct optimal 3D points from the tracked feature table. These 3D points 
were then projected to the images to compute the projection errors for the whole 
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sequence. Since the tracked feature table is the same for all methods, the 
method that produces good motions will have a small projection error. The 
number of 2D points in the tracked feature tables are 7990, 26890, 1 1307, and 
6049 for the STN31, STN61, DYN40, and FRG21, respectively. The results of 
this analysis are provided in the projection error and running time table shown in 
Fig. 10. The projection errors are displayed in the top of each cell of the table. 
The error used here is the root mean square error in pixels. The running time (in 
seconds) of each method is displayed under each corresponding projection 
error. The numbers below the sequence names are the estimated average 
rotation angle between two successive frames. For example, the rotation angle 
between two successive frames in DYN40 is only about 2.89 degrees. 

Several observations can be made from the projection error and running 
time table. As compared with the other two local processes, the proposed 
method is in general more accurate. It is slower than Method I but is much faster 
than Method II. For the DYN40 sequence, the relative rotation angle is small and 
camera intrinsic parameters are less accurate than in other sequences. Method 
I failed in the middle of the sequence because errors accumulated from the 
previous frames became too large to obtain usable 3D reconstruction for pose 
determination. Our method, however, can significantly improve each local initial 
guess (from three views) from Method I and give a reasonable result for the 
whole sequence. This demonstrated the robustness of our method. When 
compared with the classical bundle adjustment algorithm, our method can give 
very similar results. This is true especially when the rotation angle is relatively 
large. Similar observations can also be made by inspecting the visual 
comparison results from Figs 6A through 6D, Figs. 7A and 7B, Figs. 8A through 
8C, and Fig. 9A through 9C. More importantly, our method has almost linear 
computational complexity with respect to the number of images inside the 
sequence. This can be seen from the STN31 and STN61 experiments, where 
the running time was doubled (5.250/2.578 = 2.04) when the number of frames 
increased from 31 to 61. On the contrary, the classical bundle adjustment 
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algorithm was almost 8 times slower when dealing with a sequence only twice 
longer. 



4. Additional Embodiments 



While the invention has been described in detail by specific reference to 
preferred embodiments thereof, it is understood that variations and modifications 
thereof may be made without departing from the true spirit and scope of the 
invention. For example, there are two more sets of point matches between the 
each triplet of images in the input sequence not considered in the foregoing 
description. One is those matches is only between images 7 ; . 2 and However, 
this set of point matches does not contribute to the estimation of motion M„ and 
so provides no additional useful data. The other unconsidered set of point 
matches is those existing only between 7,. 2 and This latter set of point matches 
can contribute to the estimation of motion M„ and if considered could add to the 
accuracy of the estimate for the camera pose parameters associated with 
Essentially, all this would entail is additionally identifying the point matches 
existing only between 7,. 2 and I t in each triplet of images and adding a term to Eq. 
(3) as follows: 



min 



M 2 > 7 },{Q*},{L,} 



M 2 2 N 2 2 ^ 

I Ilk,-<K M <> p ,)|| +S I||qa-<K M <>Q*)|| 

j=\ i=0 k=l i=\ 

0 2 



+1 IL-<km,,u| 

/=1 *e{0,2} 



(38) 



where the additional set of point matches existing only between I 0 and h is 
denoted by {(l 0 ,/ , hi ) I / = 1 . • • • . 0} and L/ is the 3D point corresponding to the 
pair point match (l 0 ,/ , hi )■ Similarly, the two-view cost function associated the 
point correspondences between k and h can be simplified in the same way 
described in Section 2.3 in connection with the two-view cost function associated 
the point correspondences between h and h (see Eq. (36)). Namely, the 
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simplified two-view cost function associated with each point correspondence 
between 7 0 and I 2 is denoted as: 



S' = : 



(?f 0 , 2 U 2 



(39) 



Thus, Eq. (37) would be modified as follows: 



mm 



M N 



(40) 
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